We present an audit mechanism for language models, with a focus on models deployed in the healthcare setting. Our proposed mechanism takes inspiration from clinical trial design where we posit the language model audit as a single blind equivalence trial, with the comparison of interest being the subject matter experts. We show that using our proposed method, we can follow principled sample size and power calculations, leading to the requirement of sampling minimum number of records while maintaining the audit integrity and statistical soundness. Finally, we provide a real-world example of the audit used in a production environment in a large-scale public health network.

As a key technology in 6G research, federated learning (FL) enables collaborative learning among multiple clients while ensuring individual data privacy. However, malicious attackers among the participating clients can intentionally tamper with the training data or the trained model, compromising the accuracy and trustworthiness of the system. To address this issue, in this paper, we propose a hierarchical audit-based FL (HiAudit-FL) framework, with the aim to enhance the reliability and security of the learning process. The hierarchical audit process includes two stages, namely model-audit and parameter-audit. In the model-audit stage, a low-overhead audit method is employed to identify suspicious clients. Subsequently, in the parameter-audit stage, a resource-consuming method is used to detect all malicious clients with higher accuracy among the suspicious ones. Specifically, we execute the model audit method among partial clients for multiple rounds, which is modeled as a partial observation Markov decision process (POMDP) with the aim to enhance the robustness and accountability of the decision-making in complex and uncertain environments. Meanwhile, we formulate the problem of identifying malicious attackers through a multi-round audit as an active sequential hypothesis testing problem and leverage a diffusion model-based AI-Enabled audit selection strategy (ASS) to decide which clients should be audited in each round. To accomplish efficient and effective audit selection, we design a DRL-ASS algorithm by incorporating the ASS in a deep reinforcement learning (DRL) framework. Our simulation results demonstrate that HiAudit-FL can effectively identify and handle potential malicious users accurately, with small system overhead.

Researchers associated with Google and the Partnership on AI have created a framework to help companies and their engineering teams audit AI systems before deploying them. The framework, intended to add a layer of quality assurance to businesses launching AI, translates into practice values often espoused in AI ethics principles and tackles an accountability gap authors say exists in AI today. The work, titled "Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing" is one of a handful of outstanding AI ethics research papers accepted for publication as part of the Fairness, Accountability, and Transparency (FAT) conference, which takes place this week in Barcelona, Spain. "The proposed auditing framework is intended to contribute to closing the development and deployment accountability gap of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity," the paper reads. "At a minimum, the internal audit process should enable critical reflections on the potential impact of a system, serving as internal education and training on ethical awareness in addition to leaving what we refer to as a'transparency trail' of documentation at each step of the development cycle." The framework is also intended to identify risks and reduce them to the lowest degree possible, as well as to map out how things that can be done differently in the future or how to respond to a failure after launch.

The rules of business have changed. Gone are the days when business leaders had the luxury of time to ponder significant business decisions and make momentous changes to their organizations or processes. Now, businesses are transacting and making decisions practically in real time, which also necessitates that businesses manage risks and leverage opportunities with speed, accuracy and efficiency. As auditors, we have an unparalleled view of all the aspects of a business -- both up-close from an operational standpoint, as well as from a larger perspective in the global business environment. This combination of macro and micro views allows auditors to be well-placed to advise businesses on possible risks.

When organizations use algorithms to make decisions, biases built into the underlying data create not just challenges but also engender enormous risk. What should companies do to manage such risks? The way forward is to conduct artificial intelligence (AI) audits, according to this opinion piece by Kartik Hosanagar, a Wharton professor of operations, information and decisions who studies technology and the digital economy. This column is based on ideas from his book, A Human's Guide to Machine Intelligence. Much has been written about challenges associated with AI-based decisions. Some documented failures include gender and race biases in recruiting and credit approval software; chatbots that turned racist and driverless cars that fail to recognize stop signs due to adversarial attacks; inaccuracies in predictive models for public health surveillance; and diminished trust because of the difficulty we have interpreting certain machine learning models.

A wide range of claims and news can be found online, from the technology "not being ready" to vendors declaring that AI is already within their tools. Ultimately, firms must decide when to invest in AI, or rather, which AI-enabled tool provides the best return on investment. By understanding how AI fits into their technology infrastructure and audit processes, audit managers and partners can most effectively choose the right solution. It's critical to understand what real AI and machine learning techniques mean, and the implications for the firm. So how do firms measure their readiness for AI and assess potential vendors? Before adopting AI, firms should understand how it will improve their business and bring new value to clients.

Most enterprises that don't use artificial intelligence only audit up to 10 percent of their spending, while companies that do use AI are able to audit virtually all their invoices, contracts and expenses, according to a new report. The report, from AppZen, a company that provides artificial intelligence technology for auditing company spending and travel expense reports, not surprisingly finds advantages to AI technology. AI is able to flag 8.7 percent of expenses as high risk, usually because of unauthorized expenses, errors in keyed-in amounts, and duplicate spend. AI also flags 4 percent of invoices as high risk, typically due to price, discount or terms that don't match the contract, inflated prices compared to market data, and duplicate spending across invoices or with T&E. "There's a lot that can go wrong in the enterprise typical organization's spend audit process," said AppZen CEO Anant Kale in a statement.

Dividing the work could pave the way for companies to automate elements of the audit process, allowing them to free up human resources to focus on improving controls and preventing fraud. "When clients decide to split a professional service, it paves the way for change in the competitive landscape, and that's what's happening in audit at the moment," said Fiona Czerniawska, co-founder of Source Global, which surveyed 150 executives in the U.S. and U.K who are involved in the selection of external auditors. "People are already starting to act on this." Fifty-nine percent of executives said technology firms would gather data faster and at a lower cost than external accounting and audit firms, the report said. Sixty-one percent said technology firms would do a better job of automating financial processes than these firms, according to the report.

Technology continues to change society at a rapid pace, and accounting and auditing are by no means immune. New technologies are increasingly able to mimic human activity, taking on repetitive tasks more quickly and accurately than people can. The authors provide an overview of the ways in which robotic process automation may change how the profession operates, with a particular focus on the area of revenue audits. Auditing has historically incorporated many computer-dependent tools and processes, which were often interlinked by many manual steps and keystrokes. A new set of overlay software has emerged, however, that combines these disparate actions into a single smooth automated process. Robotic process automation (RPA) uses these new software tools, such as those offered by Blue Prism or UiPath, to transform a still somewhat handmade audit process into a more assembly-line audit process.

Deloitte's award-winning artificial intelligence platform continues to innovate financial statement audits by using advanced machine learning and natural language processing to extract key information from large volumes of audit evidence. FEI Daily spoke with Will Bible, an audit partner at Deloitte & Touche LLP, on innovating financial statement audits with artificial intelligence and how it will impact the world of finance, accounting and auditing. Will is presenting at this year's Current Financial Reporting Issues Conference, November 14-15, 2016 in New York City on the topic. Will Bible: To achieve automation and ubiquitous data analytics, you need data standardization. There's been a lot of progress on digitizing information, and automating processes around that digitized information.